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Challenges  in  DoD  Acquisition 


GAO-09-362T  -  Actions  Needed  to  Overcome  Long¬ 
standing  Challenges  with  Weapon  Systems 
Acquisition  and  Service  Contract  Management 


Performance  oe  the 
Defense  Acquisition  System 

2013  Annual  Report 


•  “costs  ...  increased  26%  and  development  costs  increased 
by  40%  from  first  estimates” 

•  “programs  ...  failed  to  deliver  capabilities  when  promised 
— often  forcing  warfighters  to  [maintain]  legacy  systems” 

•  “current  programs  experienced,  on  average,  a  21 -month 
delay  in  delivering  initial  capabilities  to  the  warfighter” 

Although  DoD  is  the  largest  acquirer  in  the 


world,  acquisition  troubles  remain 1 

•  201 1  M DAP  RDT&E  cost  growth  (mean)  84% 

•  201 1  M DAP  Procurement  cost  growth  (mean)  28% 

•  Effectiveness  (1984-2011)  89% 

•  Suitability  (1984-2011)  72% 

•  Nunn-McCurdy  breach  rate  from  1997-2011  31%  1 


United  Slates  Government  Accountability  Office 

GAO 

Testimony 

Before  the  Subcommittee  on  Defense, 
Committee  on  Appropriations,  House  of 
Representatives 

Far  RsIusd  cm  OallYcrr 
Expected  at  1:30  pm.  EST 
Fctruxry  1 1. 2006 

DEFENSE  MANAGEMENT 

Actions  Needed  to 

Overcome  Long-standing 
Challenges  with  Weapon 
Systems  Acquisition  and 
Service  Contract 

Management 

Statement  of  Gene  L  Dodaro 

Acting  Comptroller  General  of  the  United  Stales 

A  GAO 

GAO-09  362T 

“Performance  of  the  Defense  Acquisition  System  2013 
Annual  Report”  Table  2-3,  page  34J 
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Root  Cause  of  Poor  Program  Performance 

Inadequate  Systems  Engineering! 

•  Finding  from  Performance  of  the  Defense 

Acquisition  System  2013  Annual  Report 

-  Dominant  root  cause  of  MDAP  Cost 
Growth 

•  Finding  from  GAO-09-362T 

-  "...  managers  rely  heavily  on 
assumptions  about  system  requirements, 
technology,  and  design  maturity,  which 
are  consistently  too  optimistic.  These 
gaps  are  largely  the  result 
of  a  lack  of  a  disciplined  systems 
engineering  analysis  prior  to  beginning 
system  development ...” 
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MDAP  Cost  Growth: 
PARCA  Root  Cause  Analysis1 


Dominant 

10  of  18  Poor  management  performance 

(56%) 

*  Systems  engineering 

*  Contractual  incentives 

*  Risk  management 

*  Situational  Awareness 

5  of  18 

Baseline  cost  and  schedule  estimates 

(28%) 

•  Framing  assumptions 

4  of  18 
(22%) 

Change  in  procurement  quantity 

Infrequent 

1  of  18 

Immature  technology,  excessive 
manufacturing,  or  integration  risk 

2  of  18 

Unrealistic  performance  expectations 

1  of  18 

Unanticipated  design,  engineering, 
manufacturing  or  technology  issues 

None 

Funding  inadequacy 

1 .  “Performance  of  the  Defense  Acquisition  System 
2013  Annual  Report”  Table  2-3,  page  34) 


Why  Do  We  Fail  to  Utilize  Good  SE  Practices? 


It’s  difficult  to  justify  the  costs  of  SE  in  terms  that 
project  managers  and  corporate  managers  can  relate 
to. 

•  The  costs  of  SE  are  evident 

-  Cost  of  resources 

-  Schedule  time 

•  The  benefits  are  less  obvious  and  less  tangible 

-  Cost  avoidance  (e.g.,  reduction  of  rework  from  interface  mismatches) 

-  Risk  avoidance  (e.g.,  early  risk  identification  and  mitigation) 

-  Improved  efficiency  (e.g.,  clearer  organizational  boundaries  and 
interfaces) 

-  Better  products  (e.g.,  better  understanding  and  satisfaction  of 
stakeholder  needs)' 


We  need  to  quantify  the  effectiveness  and  value  of  SE  by 
examining  its  effect  on  project  performance? 
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The  2012  SE  Effectiveness  Study 


Purpose 

•  Strengthen  the  business  case  for  SE  by  relating 
project  performance  to  the  use  of  SE  practices. 

Method 


INCOSE 

International  Council  on  Systems  Engineering 


tm 


•  Contact  development  projects  using  the 
resources  of  NDIA,  AESS,  and  INCOSE. 

•  Survey  projects  to  assess  their 

-  SE  activities 

-  Project  performance 

-  Degree  of  challenge 


STRENGTH  TIIROl'GII  INDUSTRY  A  TECHNOLOGY 


Software  Engineering  Institute 

Carnegie  Mellon  University. 


•  Process  responses  to  identify  statistical  relationships  between  parameters. 


Survey  Tenets 

•  All  data  is  submitted  anonymously  and  handled  confidentially  by  the  SEI. 

•  Only  aggregated  non-attributable  data  is  released. 


Software  Engineering  Institute 
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Program  Performance  (Pert) 


The  Bottom  Line:  SE  =  Performance 


Program  Performance  vs.  Total  SE 


100% 
90%  -H 
80% 
70%  41 
60% 
50%  41 
40% 
30% 
20% 
10% 
0% 


15% 


33% 


52% 


24% 


47% 


29% 


Lower  SEC  Middle  SEC  Higher  SEC 

(n=48)  (n  =49)  (n=51) 

Total  Systems  Engineering  Capability  (SEC) 


- 

Higher 

Perf 

- 

Middle 

Perf 

Lower  : 

44  Perf  4:4: 

All  Projects 

Gamma  =  0.49  p-value  <  0.001 


Across  ALL  projects, 
1/3  are  at  each 
performance  level 

For  Lower  SEC 
projects,  only  15% 
deliver  higher 
performance 

For  Middle  SEC 

projects,  24%  deliver 
higher  performance 

For  Higher  SEC 
projects,  57%  deliver 
higher  performance 

Gamma  =  0.49 
represents  a  VERY 
STRONG  relationship 
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For  Challenging  Projects 
SE  is  even  MORE  important 


Perf  vs.  SEC_Total  (Low  PC) 


Perf  vs.  SEC_Total  (High  PC) 


100% 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 


Lower  SEC  Middle  SEC  Higher  SEC 
(n=22)  (n=26)  (n=25) 


Lower  SEC  Middle  SEC  Higher  SEC 
(n=26)  (n=23)  (n=26) 


Gamma  =  0.34  p-value  =  0.029 


Gamma  =  0.62  p-value  =  0.000 


A  STRONG  relationship  between  Total 
SE  and  Project  Performance  for  LOWER 
CHALLENGE  projects 


AVERY  STRONG  relationship  between 
Total  SE  and  Project  Performance  for 
HIGHER  CHALLENGE  projects 
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Study  Participants 


Participant  Solicitation 

•  Contacted  key  members  of  major  defense 
contractors  to  promote  study  participation 

•  Contacted  the  memberships  of  NDIA  SE  Division,  . 

IEEE  AESS,  and  INCOSE  INCOSE 

International  Council  on  Systems  Engineering 

Collected  148  valid  responses 


Please  enter  the  country  in  which  most  of  the 
design  and  development  engineering  will  be/was 
i3n  performed. 
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SE  Deployment  and  Performance 


SYSTEMS  ENGINEERING  DEPLOYMENT 


Integ  Product 
Teams 


Risk 
Management 


Product 

Integration 


Validation 


Monitor  & 
Control 


Total  SEC 
3.4£ 


Trade  Studies 


Early  SE 


Project 

Planning 


Req'ts  Dev't  & 
Mg't 


Verification 


Product 
Architecture 


onfiguration 

Mg't 


DEF 


•Non-DEF 


PROJECT  PERFORMANCE 
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Total  SE  vs.  Project  Performance 


Project  Performance  vs.  Total  SE  (defense) 
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■  Projects  delivering 
HIGHER 
performance 

□  Projects  delivering 
MIDDLE 
performance 

□  Projects  delivering 
LOWER 
performance 


Total  Systems  Engineering  Capability  (SEC) 

Gamma  =  0.57 


Project  Performance  vs.  Total  SE  (non-defense) 


/ 

A  Very  Strong  relationship 
between  applied  SE  and  Project 
Performance  for  both  Defense 


100% 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20% 


and  non-Defense  Projects 


10% 


Lower  SEC  (n=1 2)  Middle  SEC  (n=8)  Higher  SEC  (n=1 1) 

Total  Systems  Engineering  Capability  (SEC) 


Gamma  =  0.66 


■  Projects  delivering 
HIGHER 
performance 

□  Projects  delivering 
MIDDLE 
performance 

□  Projects  delivering 
LOWER 
performance 
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Architecture  vs.  Project  Performance 


Perf  vs.  SEC_ARCH  (defense) 
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Lower  SEC  (n=19)  Middle  SEC  (n=33)  Higher  SEC  (n=23) 


■  Projects  delivering 
HIGHER 
performance 

□  Projects  delivering 
MIDDLE 
performance 

□  Projects  delivering 
LOWER 
performance 


SE  Deployment 

Gamma  =  0.38 


Perf  vs.  SEC_ARCH  (non-defense) 


/ 

A  Strong  relationship  between 
Architecture  activities  and  Project 
Performance  for  Defense  Projects 

A  Very  Strong  relationship  for  non¬ 
defense  projects 
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0% 


Lower  SEC  (n=14)  Middle  SEC  (n=6)  Higher  SEC  (n=1 1) 

SE  Deployment 

Gamma  =  0.54 


■  Projects  delivering 
HIGHER 
performance 

□  Projects  delivering 
MIDDLE 
performance 

□  Projects  delivering 
LOWER 
performance 
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Requirements  Dev’t  &  Mg’t  vs.  Performance 


Perf  vs.  SEC_REQ  (defense) 


100% 

90% 

80% 

70% 

60% 

50% 

40% 
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Lower  SEC  (n=22)  Middle  SEC  (n=26)  Higher  SEC  (n=27) 


□  Projects  delivering 
HIGHER 
performance 

□  Projects  delivering 
MIDDLE 
performance 

□  Projects  delivering 
LOWER 
performance 


SE  Deployment 

Gamma  =  0.46 


Perf  vs.  SEC_REQ  (non-defense) 


100% 
90% 
80% 
70% 

A  Very  Strong  relationship  60% 


between  Requirements  activities 
and  Project  Performance  for  both 
Defense  and  non-Defense 
Projects 


50% 

40% 
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10% 

0% 


Lower  SEC  (n=1 2)  Middle  SEC  (n=10)  Higher  SEC  (n=9) 

SE  Deployment 

Gamma  =  0.6 


■  Projects  delivering 
HIGHER 
performance 

□  Projects  delivering 
MIDDLE 
performance 

□  Projects  delivering 
LOWER 
performance 
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Risk  Management  vs.  Project  Performance 


Perf  vs.  SEC_RSKM  (defense) 


100% 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 


28%  I  25% 


■  Projects  delivering 
HIGHER 


44% 


46% 


29% 


27% 


23% 


performance 

□  Projects  delivering 
MIDDLE 
performance 

□  Projects  delivering 
LOWER 
performance 


Lower  SEC  (n=25)  Middle  SEC  (n=28)  Higher  SEC  (n=22) 


SE  Deployment 


Gamma  =  0.28 


t 


A  Moderate  relationship  between 
Risk  Management  activities  and 
Project  Performance  for  Defense 
Projects 

A  Very  Strong  relationship  for  non¬ 
defense  projects 


Perf  vs.  SEC_RSKM  (non-defense) 


Lower  SEC  (n=14)  Middle  SEC  (n=8)  Higher  SEC  (n=9) 

SE  Deployment 


Gamma  =  0.4 


■  Projects  delivering 
HIGHER 
performance 

□  Projects  delivering 
MIDDLE 
performance 

□  Projects  delivering 
LOWER 
performance 
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Trade  Studies  vs.  Project  Performance 


Perf  vs.  SEC_TRD  (defense) 
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■  Projects  delivering 
HIGHER 
performance 

□  Projects  delivering 
MIDDLE 
performance 

□  Projects  delivering 
LOWER 
performance 


SE  Deployment 

Gamma  =  0.45 


Perf  vs.  SEC_TRD  (non-defense) 


100% 
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70% 


A  Very  Strong  relationship  between  60°/o 
Trade  Study  activities  and  Project  4o% 
Performance  for  Defense  Projects 

Z\j  /o 
1 0% 

A  Strong  relationship  for  non-  0% 


defense  projects 


Lower  SEC  (n=7)  Middle  SEC  (n=1 5)  Higher  SEC  (n=9) 

SE  Deployment 


■  Projects  delivering 
HIGHER 
performance 

□  Projects  delivering 
MIDDLE 
performance 

□  Projects  delivering 
LOWER 
performance 


Gamma  =  0.3 
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Summary  of  Relationships  -i 


Performance  vs.  SE  Capability 


-0.3  -0.2  -0.1  0.0  0.1  0.2  0.3  0.4  0.5  0.6  0.7 

_ I _ I _ |  i  i  i  i  i  _ _ I _ 
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Summary  of  Relationships  -2 
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Next  Steps:  Investigate  the  differences  between  SE  deployment  /  effectiveness  in 

defense  and  non-defense  domains  to  find  “transplantable”  best  practices 
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Questions  for  Further  Study 


On  non-defense  projects,  why  are  SE  activities  in  Requirements, 
Architecture,  Risk  Management,  and  Verification  more  effective  than  those 
on  defense-related  projects? 

On  defense  projects,  why  are  SE  activities  in  Trade  Studies,  IPTs,  and 
Project  Monitoring  and  Control  more  effective  than  those  on  non-defense 
projects? 

Why  is  the  relationship  between  Project  Challenge  and  Project 
Performance  stronger  for  non-defense  projects? 

Why  is  the  relationship  between  Prior  Experience  and  Project  Performance 
stronger  for  non-defense  projects? 
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SEI  -Your  Resource  for  Software 


and  Systems  Engineering 


For  more  information,  contact 
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